Skip to content

Multi-Domain#105

Merged
rafapi merged 167 commits intomainfrom
multi-env
Feb 4, 2026
Merged

Multi-Domain#105
rafapi merged 167 commits intomainfrom
multi-env

Conversation

@rafapi
Copy link
Collaborator

@rafapi rafapi commented Nov 17, 2025 •
edited
Loading

Enables simultaneous training across multiple domains (math, coding, function calling) with domain-agnostic orchestration

Architecture

Component Role
multidomain/loader.py Parses :: syntax, concatenates datasets, injects domain field into each sample
dispatcher.py Routes problem["domain"] → domain-specific rollout callable via actor.domain_rollouts mapping
domain_sampling.py Weighted sampling with adaptive rebalancing based on completion ratios to maintain target mix despite varying rollout latencies

Configuration

  actor:
    domain_mix:        # sampling weights (normalised at runtime)
      math: 0.4
      coding: 0.3
      fn_calling: 0.3
    domain_rollouts:   # domain → rollout function mapping
      math: pipelinerl.domains.math.generate_math_rollout
      coding: pipelinerl.domains.coding.generate_coding_rollout
    domain_system_prompts:  # per-domain system prompts
      coding: "You are an expert Python programmer..."

  train_dataset_names:
    - math::open_reasoner_zero_57k
    - coding::coding@train

Adaptive Sampling

DomainWeightedSampler adjusts weights dynamically: adjusted_weight = base_weight × (target_ratio / actual_ratio), clamped to [0.1, 10.0]. This compensates for domains with slower rollouts (e.g. coding sandbox execution) to maintain the configured mix in the output stream.

Metrics

Per-domain stats: domain_mix_actual/{domain}, domain_mix_target/{domain}, domain_mix_count/{domain}.

rafapi added 30 commits November 6, 2025 19:26
@rafapi
Add environment selector
cac78d7
@rafapi
Fix env launcher
df1d846
@rafapi
Adapt domains to env registry
9735130
@rafapi
Adapt domain configs
bb5e5ca
@rafapi
Collect env info
a1a02bf
@rafapi
Remove unrelated files
e4d0bc4
@rafapi
Remove backup
da43cbc
@rafapi
Remove duplicates
5b18001
@rafapi
Restore
9af7329
@rafapi
add domains
599b510
@rafapi
add coding
32eb5b8
@rafapi
Add remaining loaders
efaec65
@rafapi
Domain mix tracking metrics
1220f6d
@rafapi
update domain rollouts
158b2ea
@rafapi
refresh async llm flow
fe8e728
@rafapi
sync coding init
6f9c5cc
@rafapi
expand coding dataset
455ed42
@rafapi
remove legacy executor
795e490
@rafapi
revise coding rollouts
68072b1
@rafapi
adjust multidomain loader
981cb74
@rafapi
refresh preprocess pipeline
f7e6946
@rafapi
enhance utils helpers
73ca9d1
@rafapi
add multi domain config
38ff188
@rafapi
introduce domain sampling
2c5ebfd
@rafapi
add coding sandbox test
40cf648
@rafapi
implement verifier api
2c74b77
@rafapi
add symbolic init
cdfe57b
@rafapi
add symbolic dataset
a3c4106
@rafapi
add symbolic rollouts
7eef15e
@rafapi
remove deleted domains
cc091ac
rafapi added 20 commits February 4, 2026 12:04
@rafapi
remove redundant domain extraction from actor
a71eeb8
@rafapi
add explicit domain field to math rollouts
ada52ee
@rafapi
add explicit domain field to coding rollouts
3b0930e
@rafapi
add explicit domain field to fn_calling rollouts
41f9610
@rafapi
add explicit domain field to guessing rollouts
dbfdc3a
@rafapi
add explicit domain field to counting rollouts
24a2b1d
@rafapi
add explicit domain field to counting tapeagent
492bfcb
@rafapi
add explicit domain field to chartqa rollouts
30ce103
@rafapi
add explicit domain field to miniwob rollouts
7ab076a
@rafapi
add explicit domain field to deep_research rollouts
d13a485
@rafapi
remove actor.domain_mix null override from base
48d15db
@rafapi
remove actor.domain_mix null override
fce7011
@rafapi
let single-env configs fall back to env_replicas
3e180a7
@rafapi
restore cfg.environment for tapeagents compat
5a94da6
@rafapi
remove math from debug config (needs verifier)
9aace39
@rafapi
revert web.yaml to main (keep environment singular)
b39e9f3
@rafapi
add math env to debug multi domain config
b8c71c4
@rafapi
add math env to debug multi domain config
dc9dc2f
@rafapi
add missing _self_ to multi_domain base
76a9a84
@rafapi
remove domain_mix from base defaults
a423d76
ehsk
ehsk reviewed Feb 4, 2026
View reviewed changes
ehsk
ehsk reviewed Feb 4, 2026
View reviewed changes
ehsk
ehsk approved these changes Feb 4, 2026
View reviewed changes
Copy link
Collaborator

@ehsk ehsk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM overall except a couple of minor things!

@ehsk
Copy link
Collaborator

ehsk commented Feb 4, 2026

Here's also a sanity check to compare pre-multi-domain (displayed as ref below, orange in the top row and light blue at the bottom) vs. multi-domain on only MATH:

Reward Entropy
GRPO
GSPO

ehsk
ehsk reviewed Feb 4, 2026
View reviewed changes
@rafapi rafapi merged commit 39830c4 into main Feb 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ehsk ehsk ehsk approved these changes

@ollmer ollmer ollmer left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@rafapi @ehsk @ollmer